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NEJI^ ENGLAND INSTRUCTION^^ TELEVISION RESEARCH CENTER 



(NvE T R E C) 



OBJECTIVES • 



/ 1. E^'rovide impartial, independent evaluation of the ef fect^ ' 
IvdnBSs of » instructional/informational television programs in 
co^mtHilcat ingr to their intended audiences, and to. conduct evalua- 
ti^bns ^.ndepe'ndent of any comraitm^t to a particular production ^ 
0..1I broadcasting function, pijblishing*' interest , or educational 
point of v\ewl • \ / /' . * • ' • 

, f 2. Devise gene.ral models £or systematic assessment of , 
'cromprehensibility arid comprehensiqii of instructional/informational 
tfelevision proq^raws By specific target audiences as a 'routine 
part of the teSvision ^produ9tionPN|)rocess." , ^ 

I 3. * Disseminate- techni^l expertise for conduct ing- compre- 
ifension and. comprehensibility assessments among\ag^ncies ^ahd 
persQnnel clDjicerned with television' product ion in education and 
niass communicatfions. ' " * X * 

\ ^ ^ \ ' • * , ' . e " 

V 4\ Contribute to the advancement of teleyi^on as a-majfer'. 
learning resource in contemporary education-. 



R ATIONALE - ' 

It is now almost universally acknowledged that broadcast 
television is one of the m'ost^ potent forces in the lives of 
American children • IjTevertheless , television 'occupies only-a 
minor role in contemporary schools, despite the tremendous power 
the medium has demonstrated in shaping kno^^ledge, ideas, attitudes, 

and values. . ' \ 

. ^ - 

In the early futu^'e n^w technologies such as cassettes and 
cable promise* to make the dissemination ^pf educational tele-? 
vision programming, even more flex;Lble and widespread** than it is- 
today. These .technologies^ could become the*crucial link in 
actualizing -ma jbr new developments in education such as truly 
individualized instruction, expahdirig home-based instruction and ' 
open universities, and making ed.ucation more relevant to a com- ^ 
plex society, ^ 

Howeyer great these potentialities are, the impact: of tele- 
vision on the schools will continue to be minor without .satis- 
fact;ory demonstration that individual instructional television 
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progT^ams can be .trusted 'to communicate specific curricular and • 
transcurrfcular messages in ways that viewers clearly comprehend, 
and in -ways that clearly affect th^ir learning progress. 

This probldi^ of ocomprehensibility of learning materials 

points to an utterly astonishing incQngrui-ty in Ameriqan ecjuca- 

tion: enormous resourc-es and sophist i'cated^ technologies, are 

devoted to the systematic, almost obsessive measuremsnt of the 

students* performance, Y^t. no gystemat ic , widespread measurement 

whatsoever is devoted .to evaluating the performance of the 

educational materials from which the students are expected to 

learn, . ^ ' ^ . . ' 
. * • t 

The absence of comprehension and .c'omprehensi^ility assess- 
ment with instru.ctional teLevision is especially conspicuous. 
Recent. research shows that informational television programs can- 
not, be counted upon to communicate effectively to audiences of 
students anr) others to whom thay are directed. Viewers often 
s4mpfyao.not understand importajjt as^^cts of the message in the 
%;ays^iri which they are presented^ IfJzJ^y cannot -understand,, 
they-4gt^hot b§*'^xpee:ted' to learn. ' ^ ^ 1 ^ { 

Access to stable, reliable informat'ion^ about the ef f ect iVenes j 
of instructional television programs is the crucial, mj.sslng 
element in expand ing -the power of, television as a ma.jor force in 
mpdfern education, . ^ . ' ^ 

. When presently- availably, methods become accepted in common 
use for^attaiaing information about the comprehensibi lit y of. 
educational television, educators" who make television poM^ies 
and procJucers v;ho make television programs will at last have an 
ob^tive *v;ay to make and justify the very large investme"-nfe5 that " 
the wise use of television requires ;f ■ - 
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Immediate: * 
s ^ • - 

V* 

1,^ Assess comprehension and comprehensibility of indi- 
vp.dual instEUCtional -televisions programs and program 

series • * ^ • - v r 

: 

' ^ a) among represent^J: ive samples 'of' target audiences 

b). under nat uralj.st ic cl^assroom and ot^er 4p^rn ing 
coridftions representative .of actual irfst ruct iohal 

^ envi'rortments • • ' \ 

« • 

- Conduct these assessments with completed programs and 
portions of programs as they Are in production.. 



i ^ ■ O ' • '• • 

' ' ° ■ ■ • ' ■' ; T. . . ' 

Provide comprehension and comprehensibility assessment' ♦ 
findings'to producers, distributors',, and educational " 
'aqencies Ithat' uti,lize the' programs. 

ID evote.v part icular effdrt to asse.ssing. comprehensibility 

and communicative effectiveness of programs dealing with 

high priority^ "txanscur.ricuiar " tv. cs of special social 

significance '(accident prevention, i...aith and hygiene, 

drug abuse , 'ethni.c community relat ions , vocat ional educa- 

^ t ion-; eTcTTT ' ' ' 

* i ^ 

DevotB part icular > effort to assessing the comprehensi- 
bility and 'communicative effectiveness of* television 
material directed to audiences with special informat'ional 
needs (mentally retarded, hearing/vision impaired, pre- 
school and pifimary age, etc.).. 

Conduct bjasifc research- stud iets of visual, auditory",- a*nd 
lan'guage attributes of instructional television programs 
thatr enhances or impede effecl^ive communication to various 
^ I' J representative audiences. * r ^ " 

6.. Continually upgrade^echnical - expert ise £§)r conducting 
assessment ^procedures.' ' ^ ' ^ . , 

Long Range : » , . " » 

7.. .Conduct basic research on children's acquisition of media 
, literacy and on the comprehension and coriiprehensibi 1 ity 

characteristics^ of d-istinctive target audiences , and specif^t: 
' programs . * \ - - ^ 

8;^ Refine, me.thods for disseminating techniques and finding^^ ' 
generated by this^ Research Center to producers and edu- 
cat ional aqencies th^t utilize .television materi9i-].s , 

9. Develop a professional staff and consultant groups informec3 
on variou.s'^ aspects .of the applications of television 
technology in education uncommitted to 'any production, 
^sale§, or commercial interests. . . ' . 

10. ^Provide d&ta-based professional consulting services. to 
' 0 pjrpduce£»s , broadcasters, an(3 edupatotfs on various aspect^ 

of television in education. ♦ . • ^ - - \ ' . . ' 

• li . Xont r-i'bute • t.o,and exchange inf'brmaf ion with national and 
' ' international agenci^^* concerned^ with, the enhancement of 
effective televisioxk and 'mas^'' communications in all ar^eas 
6f education and' the dissemination'of >l,nforma4:ion. \ 
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MAXIMIZING COMPREHENSION OF- INFORMATIONAL FILMS AND VIDEOTAPE ; 
Formative Evaluation During Production 

The purpose of this itiemorandvim is to provide a brief descri^ 
tion of ani operational method for' assessing viewers' comprehen- 
ii6n of" informational films and videotape. 
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the ^objective conducting such assessment is to determine 
\fie communicative ef f ect ivene'ss of these fil^ms and videotapes, 
and tb upgrade^ thejn sp a&- to make sure that information is 
presented in ways\^4t targef audiences most readily understand . 



The un'iqqe feature of formative evaluation is that these „ . 
assessments are conducted asj a part of th^ production process. 
. Henc'e it is possible to generate, feedback on communicative 
effectiveness tg op^^^-i f^ »ri' ^. id 'i ^nt^es while the tape's and films 
a re actual l y in production — when- revision' is most economical. 

In 'the interests of brevity and emjjhasis upon the opera- 
tional plan-, this description ,abes,not disctJBs' the reasons. why ^ 
assessing comprehension' is important 'for improving the jnfprma- • 
tional effectiveness of'; films and videotapes. These reasons, are ^ 
set forth at length In other memor.apda and reports (see Appendix) 
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r A pn-vato urban Un,vcr$,ty. cuppono<^ by aumm. nrands. loundat.ons and ccrpor.u,ons 



It is sufficient here simply to sum up two principal paints 

/'' 

* * 

lo Comprehejnsion and comprehenffibility of informational ' ^ 

films and videotapes are the fundamental prereqi^isites 

<f ^ - • 

^ ' if they are- to have any communications benefit whether. . 

for Instructional teaming^ expanding awarenesS^o^^new- 

f ields , "product descriptions, personnel twining, pub.lic 

information/ or any other purpose'* A(3diences cannot ^ 

• . respond positively* to filn\ or videotaf>e ''material they 

« ^ J ' . . \> * " , t 

\ do snot^adequately understand. > Recent research strongly 

•j" , ^ ' ^ • : r. ' - ^ . 

sugcrests. that these materials are of teft. not, adequately ^ 
\ undef3t6od ♦ • " " ' ' * \ 

2. The formative s€age of/ f ilm aind videotape product ion • is 
the critical point for conducting systematic -evaluation 
of the communicative effectiveness of 'programs in .reaching 
the audiences* for which they are intended. 

' • . • ■'•■■■■■/ 
At this stage rfys'tematic* evaluation of audience com- 

^ * * ) 

^preliension helps produbers^ eliminate sojprces of misper- 
ception and outright errors of understanding by determining 
the extent to which samples of target audiences actually 
"^derstand program content. When these .assessments are 
dahe during the production process revisions can be 
'prepared and integrated into th^ firial program at minimal 
. . cost. » • ' ' , " 

. A recent policy statement by Keitb Mielke, Professor, of Te^le- 
communications, Indiana Uni\^rsity,. strongly substantiates the ( 
importance attributed *to, formative assessment during the prodWtion - 
period.' ^" . 

Production of television "programs Ond films]] ip' a complicated 
process in which innumerable assumptions about aud^iehce ^ 
attention, bomprehenslon, * and other reactions mus/t be made. 
Expert/ise in the production process is^needed s^- that^ a fair 
. ^hare of these assumptions will turncoat to be/correct. ♦ 



r experience shows, however, that expert judgments alone... 

'^re frequently wrong . Ultimately, there is no subs^titute . 
for k try-out' pf the program with representative audience ••* 
' ■, members-, 

I Despite the Sjariousness of problems of comprehension ' in com- 
munication by films and videotapes, no effective means have yet 
become established to Incorporate formative research as p^>^t of 
the normal production process within the general cost structure of 
producing modest-budget informational filmland videotape programs. 
Hencei even as the critical necessity for formative research on 
program effectiveness becomes recognized, methods for performing ^ 
this crucial part of the production process on a routine basis i 
remain to be devised. 

■ ■ " ■ ■ y 

Recognizing this gap, investigators at the Psychology Depart- , 
ment of the University of Hartford have .devised newy methods with 
which to measure comprehension and comprehensilDllit^y of film and 
videotape presentations. Collaborative work , is now in progress t 
with Connecticut Public Television to incorporate these test methods 
•into the production process of informational programs. 

r 

This kind of formative research is not. the only element 
needed to bring about' a major improvement in the quality and effective 
ness of informational films and videotapes. Yet it does Constitute 
the s i'nq le' mos t powe r f u 1 step that can be taken now, with reasonable 
resources, to produce 'a substantial 'increase in the effectiveness 
"and accountability of such programs., . , 



Operational Method 



What follows is an overview of the operational method for 
conducting formative evaluation of informational film and video- 
tape programs as part- of the production process.' 



•'•Keith W. Mielke, Decision-oriented research in school television 
Public Telecommunications Review, June^ 1974. 
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1 ^ ^ ^ 

Meagurement Techniques , ^. ^ • ^ 

The first requirement is to determine the level 'of' audience 
media literacy and assess specif ic , attributes of informational ^ 
films, and videotape programs that ^ are more and less successful ' ' 
in^ communicating to target audiences.. 

Three distinctive techniques have-^beeh devi'sed at the 
University of Hart|ord for assessing. aspects of , audience 
comprehension of informational film and videotape presenta- 
♦ 'eions. When these techniques' are combined' with .orthodox, 
survey and testing procedures they constitute a^powerfuL ^ 
method fbr; determ^ing. respojisei to specif ic .aspects of pro- • 
grams by carefully choken segments of the target audience , 
to which the programs are directed, . ^ 

These techniques lend themselves to efficient, re^r^; 

. latively '^rnass prodUction|^ assessment procedures , making 
it possible to identify the communicative effectiveness of 
individual linguistic and visual program segments for dif-^' 

.ferentiated portions of sample audiences. 

For example, It has been determined with highly explicit" 
measurement data that members of a more competent* audience quite 
readily idWderstood a cutaway diagram, of a housing structure 
in a program about homes around the world, while members of a 
less competent audience had gre^t difficulty comprehending ^ 
exactly the s^me explanatory sequence (see VIDEOTEST #1A, graph, 
test item 6) • ^ ' ' ' ' ' ' 

These distinctive techniques involve the concerit of the 
informational unit , an efficient workbook procedure , and 
stop-action, testing . The information unit gives great flexi- 
bility and opportunity for fine-grained analysis in determining 
aspects of the program to be selected *for assessment. 
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A 15-minute nutrition program now under analysis has been 
found to have. 360 separately identifiable information units, 
an average/"^; 24 information units per minute/ ' Neither in- 
fojfihafcional films, videotapes, nor any other instructional 
material is generally analyzed in terms of such discrete 
constituents of communication — but this is the level at 
which communication succeeds. or fails , and this is the level 
at which comprehension analysis" *iriust be undertaken, 

\The workbook procedure makes it possible to test as many 
'people as may be viewing .the program in an actual presenta- 
'SA;ori situation, Th(> method of stop-action testing fallows 
tlif^^ assessment of comprehension of individual infonnatioH *u.iits 
y ' by? large numbers of viewers within the immediate i/ime frama 
of the presentation of information being evaluated. Hence/ 
t)1e procedure ^consists more nearly of a "pure" test of com- 
prehension, relatlvel^ unconf ounded^ with issues such as short 
term and "long ter;m memory. . 1 ^ 

\ / ^ _ 

The reader must be?I\r in mind that thisj overview does ^ 
not and iS not intendW to deal with the intricacies of com- 
plex learning and cognitive integration •\ Comprehension is 
, def iTied here primarily in terms' that Carrol'l refers to as 

"plain sense comprehension," in which the receiver appre- , , 

^ . \ . ■ ■ ' ! 

hends the surface information in messages I vithout regard to 
the involvement of more elaborate inferential processes, 

♦ I , I ' ■ . 

I 

The techniques noted abov^ are appropriate for assessing 
viewers' plain sense comprehension- of informational^ film 
and videotape presentations. In addition; more complex aspects 
:>f comprehension are evaluatedywith highly detailed struc- 
tured and unstructured interview procedures, general knowledge 



2 ^ ■ 

Joho B. Carroll and Roy O. Freedle^ Eds., Language Compre- 
hension and the Acquisition of Knowledge ^ Washington, D, C. ; vTH. 
Winston and Sons, 1972. \ ^^-^ 



surveys, and tests of larger spans of information integra- 
tion>> ' , - 

Measureinents of attention such as the Kohnstaxnm-Leye^lt 
♦technique^ adaptations of the Palnier visual distractor 
techniqae, thiT^ried lander PLAYTEST technique, and the ^ 
Priedle^nder-Wetstone audio distractor technique are also 

available where appropriate. 

/ , . * ■ 

1 ' " ^ ' ■ 

=' The combination of these" techniques gives the overalL 

evaluation program unusual potency in determining the re- 
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laij:ive degi^es of success and failure with which individual 
visual, audit^oxy, and linguistic attributes of programs I * 
cohvey their itieanihg. 7 ■ ^ 

In addition J:o employing these and related inquiry 
techniques at their present level of development, the opera- * 
tional method also calls for a 'continuous process of up- 
grading the techniques through constant use and inier^ive-' 
statistical analysis of individual sets of perfonna.nce 
j - data. ' As noted on. the flow chart (page lo) , comprehension 
* • assessment of both the target 'programs that are^ under pro- 
duction and of analogue program^' comparable to those under 
production can* be scheduled on a continuous basis. 

■ \ \ \ 

Inteqyatioh of Assessment and Production^ A 

The critical step is to use these techniques to help producers 
integrate maximiim communication attributes into new program material: 

1. within the exist in^^ general framework of production 
/ practices and production costs 

2. as these practices and cost structures prevail in modeat- * 
budget production studios oriented to educational/informa- 
tional material whether for intramural or public use. 



Particular ingenuity is directedvto formulating evaiua- 
tion measures that can be integrated into the filming 
.process' with the least ppssible disruptibn of that process, 
and at the lowest possible c^st. The next section indicates 
the jcc^xeTf ul thought and plani^ing that have already been 
devoted to providing for the, integration of the processes 
V I of producing program segments — and then revision and 

refinement as required ■J 

?' 

. Operational Procedure ; 

Methods for formative evaluation during the production of 
informational films and videotapes are^ presented here in two 
w^ys.: by means of capsule' statements -describing jthe operations - 
in the general order in which they aire performed, and as a '\ 
floW chart 'showing in graphic form the interrelating functions 
of. the operations.*. This description explains the actual working 
relationships and specific tasks presently operating between the 
•University 6f Hartford Psychology Department and Connecticut 
Pul^lic .Television in the production of a bilingual/bicultural 
infontat ional series. ' » ' • 

o{ * Since the purpose of formative research lies in the success 
of the relationship. between the assessment team and th§ production 
*team, this process is^ always sQbject^ to review and revision to 
meet the distinctive needs of individual production centers. 

I 1." Evaluation team and .production team drscuss concepts, 
objectives, and s^cific ti^chniques of comprehension . 



evaluation with subject 



matte -pecialiats at ^he outset 



v 



of developn^nt of new program series. 

2, Evaluation arid production* teams discuss formative 

evaluation and* review* a sampli^ of completed compre- 
hension evaluation studies — wi'th program materials. 



( . 
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.evaluation procedureis, and audience comprehension data 
set side by side# 



3# Evaluation team and production team review*A»?ith subject 

matter specialists the^^fpaming of the format for 
/ ^ the new program in terms of specific elements of test- 
/ able visual, verbal, and auditpry segments e^qpected to • 
be produced* Participants ident if y.^prog ram segments - 
likely to involVe distinctive <:omprehension problems rje- 
garding specific difficulties in visual or linguistic 
representation. . 

A. Production and evaluation teetms review opportunities smd 
constraints inherent in the film or videotape production 
process and anticipate liaison problems* 

5. Evaluation team select^ and production- teaia confirms ' 
existing analo^tie materials generally comparable to 
anticipated progr^©_in format, general program style, 
and target audience. v 

6* Evaluation team prepares mat^erials for assessment pro- i 
cedures involving analogous program segments and conducts 

comprehension evaluations with representative sample 

• * / 
^ ^-^a ud ience s • / 

7# production team prepares coherent, testable segments of 
new program with emphasis on material previously identi- 
fied as posing particular problems in comprehension. 
Evaluation team conducts intensive field testing .with 
target audience samples. 

8. Production team uses comprehension performance data from 
initial field testing to revise format and content in 
terms of the sample audiences' demonstrated success and 

/ : 
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"^failure;in understanding test segments. Revise decisions 
ajid completion of program are based upon synthesis of • , 
coinprehensior|jperformance data* and prodyUCtigh constraints. 

9." Evaluation team conducts extensive field testing of 
, completed ^irst program in seri^ to identify -segments 
Z"-^ , -and. attributes with greater and lesser cfegrees of com- . 

. municative effectiveness with larger samples of target— 
^ audience. /' 

10. Production team incorporates theSe more extensile '-fil^iJlngs 
on comprehension performance JLnto productrion of subsequent 
portions- of series. 

* 

11. Evaluation team works with production team to "fine-tune" . 
succeeding segments and^ completed programs with compre- 
hension spot-check assessments where, problems arise. 

12. Evaluation and production teams systematize methods with 
increasing efficiency in. subsequent productions. 

These steps may be adjusted ^s necessary for individual produc- 
tions and in association with individual production centers^ ac- 
cording to variktions in budget, personnel, plr^or experience* and the 
informational objectives for which films and videotapes. are intended. 
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Table;' . OPERATIONS FLOW CHART 



TARGET , PROGRAM ASSESSMENTS 
(PROGRAM SERIES IN PRODUGTION) 



SUPPCfRTIVE ASSESSMENT 
(ANALOGOUS PROGRAM SEGMENTS) 



PRODUCTION 
OPERATIONS 



ASSESSMENT 
OPERATIONS 



ANALOGOUS ASSESSMENTS 



- , ^ ^. ^ Cultivate' liaison. 

Select program objectives. Explain research 
-^rget audience 



Select program format 



operations 
> ^Select sample audiences^ 



Examine- and select- 
analogous programs 



Script outline, writing, p^^^^^ knowledge survey 
casting * 



Writing, production of ^ 
early pilot segments 

I : 



^.Testing of early pilot 
satments 



Revision of pilQt segments, i 

writing, production of ^ 
new sefgments . 

In ; ' P Further field testing 

Foward planning of later 
series programs 

t. . 



Completion of initial 
program ■ 



-f Full scale 'testing 
program 



Outline*, writing, .and 
production of later 
series programs— r — 



-»Fine tune testing 



Later production 



Continuous review and refinement of procedures 



ERJC 



Select .sample audiences 



irsue full ,r,ange of assess 
' ment procedure 9 t l/nfbr^- 
mation^unit analy^sis^ 
test construction 



I 

Cont inuous " analogous seg- 
me^nt assessments in 
field; continuous data 
analysis | ^ 

I ' 

Continuous analogous 
\ .^^testing ^ 



Assemble findj^ngs / 
for future production; 



. . APPENDIX * 

^ \ . * 

The following memos knd reports state vaifious viewpoints 
on in4)roviTig the effect Lveriesa of educational/informational 
films and video presentat iorts by means of "fdrmative evaluation 
and research on audience comprehension. . ^ , 

These" methods also ,hi(ve applications to all - forma, o.f . 

', . • V- ■ ■ ■ - 

film and video, presentations that are intended to conVey specific 

information to*, specific audiences. 

• . ^ ' ' I, . ' ■ : ■ ' ■ '■ 

Ve invite! questions/ and discussion on any applications of 
these procedures to problems of cbnbuhicati in education 
business, and ii^vSustry/' - ' y 

Friedlandbr, B. Z.., Cohen, de Lara, H. , & Hesse., G. W. Systematic 
M^asufemetit of Primary School Children's Comprehension of' 
Classroom' Educational Television at Three Grade Levels. 
.* VIDEOTEST #1, Laboratory Report Al 23^, 1972. ^ ^ ^ ' 
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9 
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Program of Investigation. March* 1974. * , . 

& Wetstone, H. 'S. Effects of Informational and Cartoon 



* Program Format., Musical Distraetors, and Ag/on ^Childrei^^^ 

\ Listening to Televisioh Soundtracks. VIDEOTEST* #5 , Sumi^^r, 
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Title 

^Friediander, B.Z.l Cohen de Lara/H. , and Hesse, G.W. Evaluating the 
communicative effectiveness 'of primary grade level educational televi- 
sion presentations. ^ » • ^ • , . 

f Problem/Purpoae 

Vo. systematically evaluate the communicative effect ivenes^$^ of primary 
tfrad'e level ^ETV. presentations by elid'iting objective respc^ijses to ques- 
: ions embedded in the program material from classroom-sized\groups of 
bi].dren with minimal adult supervisory participation. \ 

. Popu-rat ion/Locat ion 

235/ormdl kindergarten, first grade, and second grade children in 11 
groups from Bridlepath and Chartet Oak Schools in West Hartford, Connect- 
iciit. The research was conducted, during May and June, 1972,. 

t f • 

I ■ ^.Abstract 



Audience response./ -opportunities werq integrated into an ETV program tar- 
getQdf for ^primary grades. Special , instruct ions were also provided so 
that the childreji's respoi^ises woyld have the desired format. 97% Qf the 
2,350 test resppiYses were scorable , indicating that the special proce- 
dure was within the capability of primary gfade pupils. The ' percentages 
of correct " responses ranged from 36% in a "slow" kindergarten class- to. 
95% in an advanced second grade class. The differences between grades 
were significant (p<.03) in^all cases, indicating group differences in 
communicative effectiveness of the TV presentation at three grade levels. 
Differences 15 correct responses across test' questions, were considered 
indicative of 'the relative ef fectivei^ess of different* s.'egments of the 
TV program. ' . \ v 

Interpretation/Practical Application 

The results indicate that objective 'responses to test cjuestions integ- . 
ranted! into an ETV program can be elicited from groups of primary age 
children in a typical" classroom situation. The' responses can be' used 
to evaluate the communicative effectiveness of ?;tV presentations to. 
large groups, or to evaluate an individual child ^s receptive communica- 
taoh competence. *' v ' - 

• ■ . ■ . \ ' ■ 

, Comments - » • . ^ ' 

•this technique is a promising tool for systematic evaluation of TV in- 
strud^iona.1 materials intended for primary age children. , The technique ; 
also offers substantial value in aiding the development of receptive com- 
munication skills in normal phi Idren, or those whb reach school age with 
.poorly developed skills' in English usatfe . 
♦ 
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INFANT/CHILD LANGUAGE REgEARCH L/^BORAfORY 
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*• 

Research Cbver Page . 
Title 

Staff Report - Measureinent Method for. Determining Preschool Children-^'s 
Comprehension of Early Education TV Program Content. . . ^ 

Prob lem/Purpose 

' ' ' * « 

The goal of seeking to enhance yourig children's cognitive and, affective 
development via TV prpgraimning has become an established reality.. How- 
ever', the effectiveness of individual program segments remains in doubt 
in the .absence of iinethods for obtaining objective measures of young 
.children's understanding, of specific program content . 

The purpose of these studies is to refine a new metliod for measuring 
young children's' comprehension of visual and linguistic information in 
tUitiohal TV programs.- , ■ • 
■ , • • . ■ Population/Location 

Study #1: Forty-two 4- and 5-year-olds in" 'ary^Ea stern urban Head Start 
Center. Other studies in progress. 

Method 



The children pa'rticipated in pairs in k/small room adjacent to their 
■ iegular classes. Each pair watched a /idectape with .-ur J-mJ-.i-Lt: 

SShts selected -from the Mos^ Impo rt a n/ ^ Person series. Each segment was 
.followed immediately by a set of binary choice comprehension O^^stions 
which the children answered by marking individual J^^, 2 

tions were designed to tap the children's ability to understand the ac- 
tion anlcornprehend concepts based on visual and linguistic information. 

None of the children had any difficulty handling- the ^"^I^S^^rKntJaef' 
The data cbnsisted, of workbook scores computed in terms of percentages 



correct . ^ 

• ' Results 
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Results to date of this continuing 'study take tw^vforms— hard data,., and 
the informed impressions of the children ' s .comprehension by an experi- 
enced examiner. Scores indicate, that theo'childr^n;s comprehension ap- ^ 
Dears to be limited to recognition of superficial action and familiar 
I'facts. They seem unable to take up inforihation unsupported by.high de- 
\grfees of redundancy. Also .present ing difficulties are . inferences about 
\feelinq tone and about continuities within seri'al actions. In the ex- 
Wner's opinion, the melodic and rhythmic eleiment in songs preempted 
attention from linguistic information contained in the lyrics. 
Methodological factors requiring further refinement involve framing and 
presentation of questions. 

Interpretation/Comment • - . ' . 

It is apparent that the'se children do not' absorb large elernents of the 
cognitive and affective content of the program sequences. 
Far more extensive research is required to refine the procedure, extend, 
/to more diverse populations, and ider^tify specific variables of content ai 
presentation that can lead to enhanced comprehension and effective commu- 
nioatlon. , ' ' ^Q. ' . . 
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Research Cover Page . 
Title 



Fr ied lander, |B*Z» and Wetstbne H.S. Effect? of informational and' 
cartoon program format, music'al distra6tors, and age^ on ^ children 
listening, to television sound t^racks • , /• • ' 

^ Problem/P.urpose . • 

Television programming appears to' be an effective means, of communica- 
tdng to young .children. Actually, however, very litt.le is known about 
how/well /y'arious programs /communicate, or about the effect of numerous 
varj-ables (children's age, program format, linguistic content, etc.) ' 
on that communication. / . ^ ■ s _ ^/ 

The purpose of this study was to investigate young children's listen- 
ing behavior to three ^representative children^'s television progrdttis , 
given a choice between the natural TV soundtrack and a musical dis- 
€ractor. • * ,/ • ' ' \ 

0 ' , Population/Location \ 

Sixty-eight children, 42 girls and 26 boys, patticipated in the study. 
All were enrolled in kindergarten through third grade classes in a 
public elementai;y school located in a working class . neighborhood . 

\ ' ' ' ' 

. v Methocj^ » 

' * * ^ • 

.The television programs consisted of (a)/;a nationalli^ familiar ^ , 
children's TV personality explaining about steam, (b) ..a well known 
popular car.toon featuring a &kit about suroitier camp^ and (c) a tui- 
"\ional cartoon abqut dental care. All three programs were approxi- 
mately thr4e minutes -in length, and were shown consecutively with a' 
20 second ^break in between. , Half of the children watched the P^^o-^ 
grams in' an a-b-c order, t,he other half in a c-b^a order • 

r 

The musical distractor consisted d£ a continuous 10 minute cassette 
loop. of popular selections (all about a^minute in length) such as 
might be encountered oh a local radio station. , 

.1 * * 

The children came to the television room, close by their regular 
classrooms, in groups of two\ They were given a pair of earphones 
and a switch box to hold in their laps**' The investigator explained 
that there v/ould be 'music playing over feheir earphones at all times, 
anja that in order to turn off the music/'and turn on the television 
so'un^ they simply had to hold down the swit'ch on their switch bpx* 
(Thfe visual component of the television was continuous.) The 
children were told" that they could listen to the music or the TV, 
and that they were free to switch back and forth whenever and as 
m^ny times as they wished ♦ A brief demonstration program assured 
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that the children, understood the procedure and/ were comfortable with 
their earphones and .switch, / 



Results 



The data consisted of (a) the number of seconds each child spent 
listening to each program, ^ (b) the number cif switch responses to 
each program, and (c) a mpment to moment o^-line readout' indicating/ 
distribution of switch responses 'within program time* 



A detailed analysis of the data provided /the following: 

1. Very highly significaht pro|l:arn effect (p ,001) r- the children 
clearly indicated their program preferences -in terms of the percent- 
ages of listening response time for the television soundtrack or the 
musical distractor. The distractor was least distracting during the 
well known cartoon program; there were no significant differences 
between the other two programs, ^ 

2. Highly significant age effect ,(p .01) - the second and third 
grade children demonstrated more decisive listening preference for the 
most popular cartoon program than did the first and second graders, . 
At the same time, however, the older children were more effected by 
the- musical distractor. They listened to the TV soundtrack only 55% 
to 78%' of program time as opposed to 75% to 90% of the time for the / 

. younger children, 

3. Significant Gradex Ordgr Interaction - the kindergarten (p ,01) 
and the first grade (p ,05) ^responded .to the order of 'the programs 
as well as to the programs themselves. The kindergarten children, 
especially, spent less time listening to the TV for whichever program 
came first. This.oappeared to. be more of a problem of response adapt- 
ation than an active response to the musical distractor, 

• Interpretation/Comment ' ^ 

The procedure described here indicates that it- is possible to make, 
exact numerical measurement of children's listening responses to 
television program soundtracks. These listening responses .can be^ 
recorded so'as^to indicate exactly those portions of the. program 
receiving greater and -"lesser degrees ;of attentional listening. 

The findings suggest that children cannot be assumed to listen at- 
tentively to TV soundtracks - even to a highly popular cartoon,, and , 
even when they appear to be watching th6 visuril display. Careful 
measurement is required to determine various ractors that affect chil- 
dren's listening to TV and their comprehension of the materials to 
which they listen, 

. This procedure would appear to be especially valuable in ascertain- 
ing the effectiveness of various program formats Bnd specifics of 
program content in mobilizing children's attentional listening to 
informational television programs,^ Further research in this area 
can be expected to establish patterns of children Is attentional 
listening to TV program material in various school and home settings, 
with children of different age levels, scholastic competence, and 
l,ingutstic-ethnic orientation. 
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I Informational Narration 



Entertainment Cartoon 



lllllll In.formationai Cartoon . 



96 Listening .Responses Time 
1 00- 




Gr. 1 



Or. 2 



Gr. 3 



Fixr 1 Children's lisl!enine response time, for natxiral television 
^ sSind^?Sck bylrade^nd fco^am. A distractor consisting 
•o? non^?ele.vant music pikyed through childrenis earphogos 
whenever they did non o^^rate switch controlling access t 
natural soundtrack. j 
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<fi -Listeninp Response Time 
1, 100 



a. Informational Narration 
)b. Entertainment Cartoon 
c. Informational Cartoon -- 



V 




Gr. 2 



Gr. 3 



Via 1 Children's listening' response time for natural television soundtracki 
• ' Effects of order of presentation^. Lxx Block I order of presentation 
.of programs was. a-b-c; in Block M, order of presentation was c-o-a,. 
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Research Cover Sheet 

Title 

Wetstone, H* S. and Friedlander , . B,Z. Comprehension Evaluation of 
an* ITV 'Program: Communicative Effectiveness and Appropriateness .of 
Age/Grade Designation. • ^ \ 'S^ 

* Problem/Purpose 

( . . . 

ITV programming is available over a. wide range- of subject matter and 
grade levels • Unfortunately, however, the vast* majqrityqf program's 
are produced and distributed without regard to solid ^factual informa- 
tion as to whether the programs communicate effectively to the audi- 
ences for which they are intended. \ \ 

The purpose of this study is to evaluate the communicative effective- 
ness of an 18-minute ITV j^ogram on -nutrition targeted for the mid- 
dle grades. 

Population/Location ^ ; 

The study Was conducted in two neighborhood schools', one an affluent 
suburban public school, ^the other a working class urban parochial 
school. 189 5th and 6th'- graders' participated . 

Method 



Communicative effectiveness was evaluated by means of a series of 
-comprehension questions administered with a simple workbook procedure. 

Comprehension quiestions were generated through an item-by-item analysis 
^ the verbal and visual information in^ the program, and then selected 

onNi^he basis of relevance to principal teaching objectives 'stated by 

the, p^?qgram producers. * ' , 

As a preliminary procedure, all the children were tested on the basic 
concepts dealt^with in the program, 'fwo v^eks lat^r, the children 
watched the pro^xam on classroom television. A teacher aide presented 
the compre,hension e^qestions orally while the children marked their 
answers in the^r workbooks. For the ma jdrity .of the children, tlje 
questions were presented^^uring brief interruptions-^t appropriate 
points in the program. ABD4^ 25% of the children were presented with 
all of the questions at onceN/hen the program was over. ^ 

Results • •. 

The data indicate that the program communicated poorly to the majority 
of the children. Comprehension score means fpr iridividual classes rai^^pd 
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5roin' 6496 to 75% (see Table 1). Question-by-qubst ion analysis, how- 
ever, indiyiatps that a third or more of the chilldren faij.ed to compre- 
hend more chaTn half the information tested. Scpres for the group* who 
viewed the prog2;am without interruption were substantially lower than 
for the majority who ans^wered the questions during program interruptions, 

Low scores indicate that the children simply 'did not understand the 
program content. ^ ' - *^ 



nts generating poor comprohsnsicn suggests 

1) use of concepts 



Ana lysis, of the pzpgran^ segitvsr 

that cqmn^unicat ion failure can be attributed to 

beyond 'children's understanding" regardless, of simplif icatipn; 2). failure 
to^d^fine terms which are superficially familiar but conceptually c^lffi- 
cult; 3) -use of visual illustrations which are logically^ sound , but 
either too abstract, or presented too rapidly ti provide support for 
the appropriate concepts. ^ ^ ' * ■ ' ^" 

It is notable that one set of facts indicated by- the program prOc^ucers 
as a main teaching objective vas communicated to less thah half of the 
children (559^ failure ) . " . ' 

Comparison of preliminary, scores to comprehension scores indicates an 
average gain of 2396 of the children responding correctly (see Figure 1) 
to specific concepts. Gains wejfe^^moderate' tc> g^od on lome concepts; 
presentation of two points actually produced deficits in comprehension. 

Int erpret at ion 

The quest iori at hand is not whether ITV programming communicates, but 
how well specific programs do their job. The e\^aluati6n procedure 
reported here provides detailed information as to which segments of 
the program communicate veil, or poorly, or not at all. 'Such' informa- 
tion is clearly necessary if ITV programming is ;to fulfill its objectives 
and justify its place in* the classroom and its tremendous cost to ^prpduce 
and disseminate. * ' * 
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• TABLE 1 
GRADING THE LEARNING MATERIALS 
INSTEAD oV 



• ' GRADING THE STUDENTS 

If a' certain .widely used IS-minute ITV program on nutrition 
*-were graded* for performance in school the same way that children 
are graded 'for performance in school, its report card would- 
look like this : 



/ 



/ 

SCHOOL GROUP 



Middle Class Suburban 

» .... 

^ K t 

t 

Middle- Class Suburban 
.(n=59) ' ■ ■ 

\ 

•WdrRing Class City 
' (n=42) 

Marking Ciass - City 
- (n=48) 



6th grade 
5th grade 
6th grade 

/ 

5th grade 



"AVERAGE " 
SCORE** 

■7-5^0 



71.0 



EXPORT- 
. CARD 



71.0 



64.5 



C- 



C- 



N=189 , 



♦ ★Average^ scdres ori 28 questions testing coinj)rehension of 
, • principal visual-linguistic 'facts and cortcepts 
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Formative" Evaluation 
( Interim Report : February', ' 19T5 ) 

M U N D 0 REAL ■ 



Produced for 
Connecticut Public Television 
■by 

Marrero Productions, Inc. 
Funded: U.S. Office of Education - ESSA 

ABSTRACT 



• 'MUHDO REAL is a 30-ininute bilingual televtiion program, first of a series of 13. 

The primary objective, of the program is to contribute to the Puerto Rican child s 
sense of -pride and self-esteem by providing opportunities to. identify with the warmth 
and richness of the Puerto Rican comnmnity in mainland cities. It is anticipated , 
that the program, which is aimed at children in grades U-8, will also provide the • 
general television audience with positive views of the Puerto Mean urban culture. 

MUNDO'lREAL is about "a Puerto Rican family and their friends and neighbors. 
The children in the family, ages 9. and 11, become involved in various events at 
school and 'in the conmunity which result in encounters ■ and interactions with their , 
•parents, neighbors, and members of the community. These events are designed to 
portray the warmth and supportivehess of the Puerto Rican family, and their pleasure 
and' pride in their Puerto Rican heritage. / 

The purpose of the formative evaluation is to determine the effectiveness with 
which the program fulfills its communication objectives. ^ 

The initial evaluation was carried out by means of systematic written procedures. 
These procedures were used to assess the program's effectiveness in mobilizing 
attention, establishing emotional and attitude values, establishing. credibility of 
characters and dramatic situations, and in transmitting primary and secondary factual 
information ahout. the program. 

This phase of the evaluation was conducted with 3lO children in grades U-8 
in pii§lic and parochial' schools in the City of Hartford. The sample audience was 
more than^e-half Puerto Rican, approximately one-third black, and the remainder 
a mixed gro§^f polyethnic white children. Children, were tested for program 
comprehension Sn^ately after viewing the program by videotape in their classrooms. 
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\ This portion of * the ^evaluation provided the following observations: f 

(for numerical details, see full Interim Report) 

' ' ' ' " 1 

1. Attentions overall patters of attention approximated 90-100J5, with minor^' lags 
to 75% during points of extended monologue or dieuLogUe; attention was more v€u:jiable 
in younger than in older children, hut extremely high in general — especially in view 
of the high levels of distraction in some of the classrooms. 



2. Emotional and Attitude Values : progVam attained h'igh levels of communication 
of the family warmth and mutual support despite some difficulties among the younger 
children* The p^^ogram succeeded in. projecfting very posi4.tive feelings aboxrt / 
Puerto Rican family . cultxire and community interactions • The younger children 
tended to perceive more negative feeling than was intended to he portrayed. 1 
Almost all the children responded very positively in terms of the program's j 
entertainment values, it9* anticipated ccamnunity impact, and its effect on their 
feelitfgs toward themselves and the Puerto Rican ctilture. * / 

,3. Credibility of Characters: ■ The majority of the children easily refcogntzed all 
the major characters and identified them as probable members of th#ir own community. 
The mother role was particiLLarly well-established, even among the non-Puerto Rican 
children. Minor characters were not as easily recognized, some of the children l^ad 
specific difficulty with two characters* The two policemen in the program were , 

*not considered to be re^ilistic portrayals. It was apparent to visual observaiion 
that black children in the classes responded favorably ,to the presence of I the 
black policemaii. " / j 

U; Credibility of Situation : The plausibility of the home, the, school, |and the • 
story action were adequately established. Many of the children doubted the 
•reality of the friendly encounters ^between the school ijuid the family, and the 
.friendship between family and policemen. The „ma;)ority- of the children accepted 
the principal lines of the story action! with no difficulty at all, and could see 
itf as possibly taking place in their own\school^ 



5. Primary and Secondary Facts : This one area differentiated stroilglyj between the 
younger and the older children, and* many of the children in all grades ^ad difficulty 
understanding ,specific points of information contained in the dialoguet'' Many of 
the younger children were \xnable to follow the flashback format of 'the /narrative. 
'The children ''s difficulty with specific points of information, however, were 
unrelated to their positive response to the overall objectives of the^ program. 

Except for elements where communication faltered, this stage of the ^evaluation 
demonstrates tliat MUNDO REAL succeeds in meeting. its objectives; For f most of the 
children in this test sample, MUNDO REAL is indeed a real world. The^ children paid 
close attention and understood most of the story. They felt comfortable with the / 
characters. They recognized the settings." Perhaps most Important, -the non-Puerto / 
Rican children came away with an enhanced -sense of the humanity of the Puerto Ricah 
culture, and the Puerto Rican children gained a sense of culturaO. and persontil 
self esteem. . ' ' / ' « 

Succeeding stages of the evaluation scheduled for con^letion this Spring 
include testing of samples i;i other cities and depth interviews intended to elaborate 
qualitative values touched' upon in this interim survey. * * 
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( Table 1 



Mundo Real 



Percent of Observed Attention by Scenes Across Grade Levels 



"5th 



Parochial and Public Schools 
PAROCHIAL SCHOOL 
6th, 7th nSth 



SCENES 



(grades) 



^ PUBLIC SCHOOL 
(groups) 

90 
100 
100 

100 



liXvxug nuuni 




100 


100 


■ 100 


l&O 


100 


100 ■ 






100 * 


100 


100 


100/ 


100 


90 . 


Assembly #1 




100 


100 


100 


100 


. 90 


9.6 


vQXX^ Wxxn wouxier 




on 


TOO 


100 


100 


80 


■ 90 


, Travel^ Agency 


<• 


80 


■ 85 


100 


95 


90 


80 


DeliaVw/ Grandpa 




75 


85 


100 


.95 


90 


80 




Yk 1 


on 


00 


100 


■95 


90 




Luand Pito 




90 


90 


100 


106 


90 


100 ' 


Creche Sc%ne 




90 


90 


100 


100 


90 


90 


Donna Ines 


a; 


05 


90 




100 


QO 


90 




b) 


= 80 


90 


100 . 


95 


85 


80 




c) 


80 


90 . 


c95 


90 ■ 


90 


90 


> 

Basketball " 


a) 


80 


■ 80 


100 


100 


100 


90 




b) 


75 




90 


95 


90 


80 


Delia on Street 


9 


• 90 


90 


100 


100 


90 


90 


Opening presents 




80 i 


96 


100 


. 90 


100 


100 


Water /Window 




85 


90 


100 . 


95 


1 100 


100 


Delia v/ Father 


a) 


85 


85 


. 95 


95 


80 


100 




b) 


90 


90 


90 


96 


80 


80 


Pito w^t 




'95 


90 


100 


IGO 


90 


' • 90 


Assembly #2 




90 


- 90 


100 


lOD 


90 


80 


Party - 




100' 


ipo 


100 


100 


90 


90 




n 


= 22 


21, 


19 


31 


h» 20 . 


23, 




N=93 








N=8o : 




fluent Spanish 


id 


10 


.7 


5 


2 


23 



19 



90' / 
90 / 



/ 



100 


itDO 


90 


/90 


90 / 


. 90 


190 


90 


/90 


• "90 


100 


90 


80^ 


80 , 


80 


70 


90 


90 


80 


80 


90 • 




100 ■ 


90 


100 . 


90 i 


■90 


80 


90 


70 


90 


80. 


:9o 


90 


90 • 


100 



18 
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N«173 



N- 62 
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Percent Negative 



Percent Positive 



10 0 8a 60/ ho 20 0 20 yo 60 8Q ^'10 0 



•Perception of family support & cohesion 

A 8 Mother figure 
» ♦ 

B 7 Brother figure 



Bl8 Father^ figure 



BIO Family 



* All* Family 

Perception of school authority 

•A15 Specific principal/child 

Al6 General principal/child: 

t ♦ 

Bl6 General principal/child 

Attitude toward this program 

B21 Entertainment value 

5* 

*• 

\ 

B22 Conmunity impact 
A20 PersoneLL iirqpact 



± 




.#« 8th, 7th grade , ★ « 6th^grade 106 '80 6d kO 20 3 20 hb 6o , 100 

■ » 5th, Uth grade > \ 

Figure 4:' -Comprehension of emotional and attitudinal values by grade. Note the unanimously 
tiigh positive attitudes toward the ^program. 
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Major Characters 

A 5 .Central figixre 



' B. 2 Central figure 



B12 Mother figure 



Father figure 



• A13 Younger Jbxother 



Minor characters 

A k Friend 



All Policemen 



A12 Grandfather 
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Figure 3: Credibility of characters ~ recognition and identification of cast, 
Note the very small divergence between grade levels. 
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Neighborhood Credibility 
• A 6 Neighborhood 

Household Credibility 
B 8 Family 



A 9 Child 

Poficeman*'s Image 
B13 

School Credibility 
BIT 



A 2 



Narrative Crisis 
B 6. 



Narrative Resolution 
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PiKure 1»: Situational Credibility children's acceptance of setting and. 

dramatic situation as realistic. Very little divergence between 
grades. 
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Comprehension of story format 
Bli 
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A 7 

Comprehension of main story- 
line 

A3 
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B 3 
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Comprehension of secondary 
story line 

B U " 



B 9 
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Comprehension of incidental^ 
• facts 

P15 





Figure 5 : 
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Percent Correct 
• = 8th, Tth grade *= 6th grade 1^ 5th, grade , 

Comprehension of principal and secondary information relating to main story , 
line, action and characters , Note high differentiation between younger and 
older children. 



